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(57) Abstract: An apparatus for swapping the content f a video signal detects the content f a video sequence in die video signal 
using extracted text and/or audiovisual features to identify the content of the video sequence. The identify of the video sequence is 
then compared with a personal profile of the user. If the content of the sequence is undesired, the apparatus swaps this video sequence 
with a desired video sequence from a known source (12, 24, 32, 34). 
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METHOD AND APPARATUS FOR SWAPPING THE VIDEO CONTENTS OF UNDESIRED 
COMMERCIAL BREAKS OR OTHER VIDEO SEQUENCES 



Field Of The Invention 

The subject invention relates to television and, more particularly, to the 
manipulation of video segments or video sequences in a television signal. 



Description Of The Related Art 

Television programs, and in particular, broadcast television programs include 
advertising commercials to at least partially defer the costs of providing the television 
programs to the viewing public. The content of these commercials are predetermined and 
1 0 usually depend on the amount of money an advertiser is willing to pay to have its commercial 
Jf$f^ broadcast at a specific time. However, no concern is made for the desires of the individual 

viewer. The viewer is left to either view commercials he/she has no interest in, leave the 
room (e g., go to the refrigerator to get a snack), or channel surf, i.e., see what programs are 
being broadcast on other channels (this can be frustrating in that, invariably, a commercial is 
1 5 usually being broadcast on the other channels). This sometimes causes problems in that the 
viewer misses the start of the program he/she was viewing before the start of the commercial. 

Consumer video cassette recorders are known which include circuitry for 
detecting commercials in a recorded television program and for fast-forwarding the tape to 
the end of the commercial. However, the television program must have been previously 
20 recorded on a video cassette. 

In another situation, television receivers now must include a "V-CHIP" which 
detects data sent with a television program concerning the content of the television program, 
e.g., the program rating, the violence content, the sexual content, etc. This allows, for 
example, a parent to control, at least partially, the content of television programs received for 
25 viewing by their children. In the event that the received program is beyond a pre-selected 
level, either a message is displayed on the screen indicating that the program (or portion of 
the program) exceeds the allowed level, or the viewer is subjected to a "blue screen" (no 
video signal for the duration of the program or portion of the program). 
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It is an object of the invention to provide a method and apparatus for 
substituting desired video sequences for undesired commercials or programs (or portions of 
programs). 

5 This object is achieved, in a first aspect of the invention, in a method for 

swapping undesired video in a specific video stream with desired video, said method 
comprising the steps of detecting a start of a specific video sequence; detecting 
characteristics of said specific video sequence; comparing said detected characteristics with 
stored characteristics of other video sequences, said stored characteristics including 

1 0 indicators specifying whether the respective other video sequences are desired; determining 
whether said specific video sequence is desired based on said stored characteristics; and 
substituting a desired video sequence in place of said specific video sequence if said specific 
video sequence is not desired. 

A second aspect of the invention provides an apparatus; for swapping 

15 undesired video in a specific video stream with desired video, said apparatus comprising 
flS# v r means for detecting a start of a: specific video sequence; means for detecting characteristics of 

said specific video sequence; means for comparing said detected characteristics with stored 
!fc characteristics of other video sequences, said stored characteristics including indicators 

specifying whether the respective other video sequences are desired; means for determining 

20 whether said specific video sequence is desired based on said stored characteristics; and 
means for substituting a desired video sequence in place of said specific video sequence if 
said specific video sequence is not desired. 

Applicants have found that in order to practice the invention, one must first 
detect the video sequence subject to being replaced. In the case of commercials, there are 

25 several known methods including detecting a change in the average light intensity of a video 
signal, detecting a change in the "activity" level, detecting increased cut rate and the presence 
of varying size text, detecting the black level in the video signal which would indicate a 
break in the program where a commercial is inserted, etc. A particular method is disclosed in 
U.S. Patent Application Serial No. 09/123,444, filed July 28, 1998 (Attorney Docket No. 

30 PHA 23,477), assigned to Philips Electronics. 

Next, the video sequence needs to be analyzed to detect known characteristic 
features. U.S. Patent 5,870,754 (Attorney Docket No. PHA 23,104), assigned to Philips 
Electronics, discloses a method in which video signatures may be extracted from MPEG or 
Motion JPEG encoded video sequences for identification purposes, and then stored in a 
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storage medium. Subsequently, a video signature of a suspect video sequence may then be 
compared with these stored video signature for the purpose of identifying the content of the 
suspect video sequence. 

U.S. Patent Application Serial No. 08/867,140, filed June 2, 1997, assigned to 
5 Philips Electronics (Attorney Docket No. PHA 23,252) describes a system which detects 
significant scenes of a video source, selects keyframes to represent each detected significant 
scene, and creates a video index for the video source. The current video sequence can then be 
analyzed and the results of the analysis compared with the stored video index. 

As an alternative, the video sequence may be examined to detect text regions 
10 in the image. U.S. Patent Application Serial No. 09/370,93 1 , filed August 9, 1 999 (Attorney 
Docket PHA 23,616), assigned to U.S. Philips Corporation, discloses a method and 
application for detecting text locations in a video signal. Automatic character recognition 
may then be used to detect 800-numbers, Domain Names, logos and product names The 
r results may then be compared with a list of 800-numbers, Domain Names, logos and product 

t> 4 . 15 names. Once the, for example, 800-number is identified, the general classification and 
l;^ 1 ' - purpose pf the commercial is known. 

^^at:iA:Sv.. : f Once the identity of the commercial is determined, this information is then 

compared to a personal profile of the user of the apparatus. For example, in the pase of a 
single childless male, commercials for diapers, other baby products or toys, would not be 
20 desired. However, that person might be interested in sporting equipment Therefore, that 
person's profile would indicate "NO" for baby commercials and "YES" for sporting 
equipment. 

If, now, the commercial is to be substituted, the alternate video stream may be 
obtained from a number of different sources. For example, the alternate video stream may be 

25 stored in a mass storage medium, e.g., a hard disk drive, a video tape, a video disc, etc. 

Alternatively, the alternate video stream may be the person's e-mail from a global computer 
network or Web pages (static or in a browse mode). In addition, in the case of digital 
television in which a plurality of programs may be available on a single channel, alternate 
commercials may be broadcast in parallel, and a desired commercial may be retrieved from 

30 this source. 
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With Ihc above and additional objects and advantages in mind as will 
hereinafter appear, the invention will be described with reference to the accompanying 
drawings, in which: 

Fig. 1 is a block diagram of an apparatus for swapping video sequences in 
5 accordance with the invention; 

Fig. 2 is a flowchart showing the functioning of the apparatus of Fig. 1; 

Fig. 3 is a flowchart showing a portion of the flowchart of Fig. 2 in greater 

detail; and 

Fig. 4 is a flowchart showing another portion of the flowchart of Fig. 2 in 

10 . greater detail. 



In Fig. 1, input video signals are received on antenna 10 and supplied to a 
: j * tuner 12. While antenna 10 is shown which woukLimply that the source of the input video 
15 signals is broadcast television, it should be understood that the input video signals may 
q^Mv, originate from other sources, e.g., cable, satellite, a global computer network, etc. The tuner 

!fe&&rr 12 tunes to a desired video signal and applies the same to a frame memory 14. An output 

, .from the frame memory 14 is connected to a, video switch 16 which applies its output to 
display device 1 8, for example, a television receiver. 
20 A controller 20, which includes an internal microprocessor, controls the 

operation of the components in the apparatus. A memory 22 in the form of random-access 
memory (RAM) is connected to the controller 20, as well as a read-only memory (ROM) 24. 
A keyboard 26 is shown for providing a user interface for the apparatus. Alternatively, an 
infra-red receiver 28 and corresponding remote transmitter 30 may be connected to the 
25 controller 20. 

A video storage device 32, for example, a video tape recorder/player, a DVD 
re-writeable device (DVD-RW), a digital VHS tape recorder/player (D-VHS), a digital video 
recorder/player (DVR), etc., is shown connected to the controller 20 for recording and 
supplying video signals. In addition, a video disc player 34 is connected to the controller 20 
30 for providing selected video programming. Finally, a modem 36 is connected to the 

controller 20 for interfacing with a telephone network 38 for allowing access to a global 
computer network enabling the sending and reception of e-mail. 

The controller 20 is connected to the output of the tuner 12 for receiving the 
desired video signal. A V-Chip/Meta data extractor 40 is also connected to the output of the 
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tuner 12 for extracting V-Chip data as well as Meta data which may be transmitted with the 
desired video signal. Finally, the controller 20 may be additionally connected directly to the 
tuner 12 for receiving programming on alternate channels. 

In operation, referring to the flowchart of Fig. 2, at block 50, the controller 20 
5 receives the video signal at the output of the tuner 12. At block 52, the controller 20 
determines whether the video signal contains a commercial. If not, the controller 20 
examines, in block 54, the extracted V-Chip data from the V-Chip/Meta data extractor 40 and 
determines whether the programming in the video signal is restricted (block 56). If not, the 
controller 20 continues to examine the video signal for commercials (block 52). If the 
1 0 programming is restricted, the controller 20 inserts alternate programming (block 70). 

When a commercial is detected, the controller 20 extracts the 
audiovisual/textual features and/or signature of the commercial in block 58, and compares 
these features or signature to a stored data-base of such features and signatures (block 60). In 
1 : block 62, it is determined whether the features and/or signature of the commercial is indeed 

i i 15 stored. If not, in block 64, the user is given the option to store the features and/or signature, 

; and if soothe features and/or signatures are stored, in block 66. If the user decides hot to store 
Wk&?&: i the features arid/or signature, the controller 20 exits the flowchart at block 72. 
r ^ l ' •>•«' . ^ At block 68, the controller 20 determines whether the commercial is desired. If 

so, the controller 20 exits the flowchart at 72. If the commercial is not desired, at block 70, 
20 the controller 20 inserts alternate programming using the video switch 16. The alternate 
programming may come from any of a plurality of sources. For example, alternate 
commercials may be stored in the ROM 24, on the video storage device 32, or on video disc 
using the video disc player 34. In the event of digital television (DTV), each channel may 
contain several video programming streams among which may be alternate commercials. If 
25 so, the alterjiate commercials may then be accessed for insertion. Otherwise, the controller 20 
may access the modem 36 to enable the user to read or send e-mail messages or access Web 
pages during the commercial break. 

The user's personal profile may be stored in the ROM 24 and includes 
information about the preferred sources, time and duration of substitution. The sources may 
30 be, as noted above, the video storage device 32, the video disc player 34, alternate 

programming from the tuner 12, or the modem 36 for accessing a global computer network 
for e-mail or web browsing. The user-preferred sources of information may be based on the 
time of day, for example, in the morning, the user may prefer substitution of another channel, 
or a personalized news channel in that the user may be getting ready to go to work and would 
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like a passive content swap. In the evening, the user may prefer a more interactive swapping, 
for example, e-mail or web browsing. 

The duration of the substitution may be based on the type of content, the 
duration of the content being replaced, and the time of day. For example, if the commercial 
5 break is 6 minutes, the user may want 2 minutes of e-mail, and a summary of the 

commercials. If the commercial is less than 2 minutes, the user may prefer no swapping. 
Alternatively, other shortened video sequences may be obtained from stored content. 

There may also be the occasion where, during a swap, the user has started an 
e-mail and does not want to be interrupted to go back to the regular programming until the 
10 user has finished the e-mail message. In that event, the regular programming may be buffered 
on, for example, the video storage device 32 or on a Tivo unit (not shown), until the user has 
finished the e-mail message. 

Fig. 3 is a flowchart showing more details of block 58 in Fig. 2. In particular, 
. iat block 80, the controller 20 extracts textual data from the. output of the tuner 12. At block 
15 82, the controller 20 then compare that extracted data with stored text and determines 
vi whether the extracted text is stored (blpck 84). If so, at block 86, -the, controller 20 proceeds to 

jfffiSf ■■)- '"■!. block 68 iii the flowchart of Fig. 2, If not, the controller 20 .then extracts the audiovisual data 
fev X .* i : in block 88 and, at block 90, proceeds with block 60 in the flowchart of Fig. 2. 

Alternatively, as noted above, the program in the video signal may include 
20 Meta data which characterizes the content of the program and/or commercials. If so, as 
shown in Fig. 4, the controller 20, after detecting a commercial at block 52 in Fig. 2, then 
examines, in block 92, the Meta data and uses this data to characterize the commercial. 
Again, at block 90, this is compared to the user's personal profile in block 68 of Fig. 2, to 
determine if the commercial is desired, and if not, the controller 20 inserts alternate 
25 programming (block 70). 

As mentioned in the accompanying claims, video signatures may be derived 
from audio clues in the video sequence. Where video signatures are so derived from audio 
clues in the video sequence, these audio clues may include energy, band energy ratio, pause 
rate, pitch, Fourier transform co-efficients and Mel Spectrum frequency co-efficients. The 
30 video signatures may be obtained from closed captioning content. 

The stored characteristics referred to in the claims maycomprise a time 
duration of the video sequence to be substituted. 
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Where the desired video sequence comprises Web pages, the pages may be 
scanned passively. Where the desired video sequence comprises Web pages, the pages may 
be scanned interactively. 

Numerous alterations and modifications of the structure herein disclosed will 
present themselves to those skilled in the art. However, it is to be understood that the above 
described embodiment is for purposes of illustration only and not to be construed as a 
limitation of the invention. All such modifications which do not depart from the spirit of the 
invention are intended to be included within the scope of the appended claims. 
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CLAIMS: 



1 . A method for swapping undesired video in a specific video stream with 

desired video, said method comprising the steps: 

detecting (52) a start of a specific video sequence; detecting characteristics 
(58) of said specific video sequence; 
5 comparing (60) said detected characteristics with stored characteristics of 

other video sequences, said stored characteristics including indicators specifying whether the 
respective other video sequences are desired; 

determining (62) whether said specific video sequence is desired based on said 
stored characteristics; and 
10 substituting (70) a desired video sequence in place of said specific video 

sequence if said specific video sequence is not desired. 

2 The method as claimed in claim 1 , wherein said method further comprises the 

steps: 

15 storing (66) said characteristics of said specific video sequence if said 

characteristics are not already stored; and 

storing (66) an indicator specifying whether said specific video sequence is 

desired. 

20 3. An apparatus for swapping undesired video in a specific video stream with 

desired video, said apparatus comprising: 

means (20) for detecting a start of a specific video sequence; 
means (20) for detecting characteristics of said specific video sequence; 
means (20) for comparing said detected characteristics with stored 
25 characteristics of other video sequences, said stored characteristics including indicators 
specifying whether the respective other video sequences are desired; 

means (20) for determining whether said specific video sequence is desired 
based on said stored characteristics; and 
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means (16) for substituting a desired video sequence in place of said specific 
video sequence if said specific video sequence is not desired. 

4. The apparatus as claimed in claim 3, wherein said apparatus further comprises: 

5 means (24) for storing said characteristics of said specific video sequence if 

said characteristics are not already stored; and 

means (24) for storing an indicator specifying whether said specific video 
sequence is desired. 

10 5. The method of claim 1 or 2 and/or the apparatus of claim 3 or 4 wherein said 

specific video sequence is a commercial (52) in a television broadcast signal. 

6. The method or apparatus as claimed in any one or more of claims 1 to 5, 

wherein said characteristics are video signatures of video sequences. 

, 15 ' ■ • 

sfe& 7. ; The method or apparatus as claimed in claim 6, wherein said video signatures 

are derived from audio clues in the video sequence. 



8. The method or apparatus as claimed in any one or more of claims 1 to 5, 
20 wherein said characteristics are textual matter contained in video sequences. 

9. The method or apparatus as claimed in any one or more of claims 1 to 8, 
wherein said substituting step comprises retrieving said desired video sequence from a 
storage medium. 



25 



30 



10. The method or apparatus as claimed in any one or more claims 1 to 8, wherein 
said specific video stream is on one channel of a digital television broadcast having a 
plurality of channels, and said substituting step comprises retrieving said desired video 
sequence from an alternate channel transmitted with said one channel. 

1 1 . The method or apparatus as claimed in any one or more of claims 1 to 5, 
wherein said detecting a start of a video sequence comprises detecting Meta data 
accompanying the video sequence. 
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12. The method or apparatus as claimed in claim 1 1 , wherein said Meta data 

comprises V-Chip data (54), and wherein said step of detecting characteristics comprises 
comparing (56) the V-Chip data with stored desired personal characteristics of the user. 

5 13. The method or apparatus as claimed in any one or more of claims 1 to 5, 

wherein said stored characteristics comprise categories of commercials and products. 



10 



14. The method or apparatus as claimed in any one or more of claims 1 to 5, 

wherein the desired video sequence comprises Web pages (36, 38). 
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